Evaluation of a Machine Learning-Based Dysphagia Prediction Tool in Clinical Routine: A Prospective Observational Cohort Study.

Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients' risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.

External Validation of a Machine Learning Based Delirium Prediction Software in Clinical Routine.

BACKGROUND: Various machine learning (ML) models have been developed for the prediction of clinical outcomes, but there is missing evidence on their performance in clinical routine and external validation. OBJECTIVES: Our aim was to deploy and prospectively evaluate an already developed delirium prediction software in clinical routine of an external hospital. METHODS: We compared updated ML models of the software and models re-trained with the external hospital's data. The best models were deployed in clinical routine for one month, and risk predictions for all admitted patients were compared to the risk ratings of a senior physician. After using the software, clinicians completed a questionnaire assessing technology acceptance. RESULTS: Re-trained models achieved a high discriminative performance (AUROC > 0.92). Compared to clinical risk ratings, the software achieved a sensitivity of 100.0% and a specificity of 90.6%. Usefulness, ease of use and output quality were rated positively by the users. CONCLUSION: A ML based delirium prediction software achieved a high discriminative performance and high technology acceptance at an external hospital using re-trained ML models.

Development of an Architecture to Implement Machine Learning Based Risk Prediction in Clinical Routine: A Service-Oriented Approach.

BACKGROUND: Patients at risk of developing a disease have to be identified at an early stage to enable prevention. One way of early detection is the use of machine learning based prediction models trained on electronic health records. OBJECTIVES: The aim of this project was to develop a software solution to predict cardiovascular and nephrological events using machine learning models. In addition, a risk verification interface for health care professionals was established. METHODS: In order to meet the requirements, different tools were analysed. Based on this, a software architecture was created, which was designed to be as modular as possible. RESULTS: A software was realised that is able to automatically calculate and display risks using machine learning models. Furthermore, predictions can be verified via an interface adapted to the need of health care professionals, which shows data required for prediction. CONCLUSION: Due to the modularised software architecture and the status-based calculation process, different technologies could be applied. This facilitates the installation of the software at multiple health care providers, for which adjustments need to be carried out at one part of the software only.

Machine Learning Based Risk Prediction for Major Adverse Cardiovascular Events.

BACKGROUND: Patients with major adverse cardiovascular events (MACE) such as myocardial infarction or stroke suffer from frequent hospitalizations and have high mortality rates. By identifying patients at risk at an early stage, MACE can be prevented with the right interventions. OBJECTIVES: The aim of this study was to develop machine learning-based models for the 5-year risk prediction of MACE. METHODS: The data used for modelling included electronic medical records of more than 128,000 patients including 29,262 patients with MACE. A feature selection based on filter and embedded methods resulted in 826 features for modelling. Different machine learning methods were used for modelling on the training data. RESULTS: A random forest model achieved the best calibration and discriminative performance on a separate test data set with an AUROC of 0.88. CONCLUSION: The developed risk prediction models achieved an excellent performance in the test data. Future research is needed to determine the performance of these models and their clinical benefit in prospective settings.

Technology Acceptance of a Machine Learning Algorithm Predicting Delirium in a Clinical Setting: a Mixed-Methods Study.

Early identification of patients with life-threatening risks such as delirium is crucial in order to initiate preventive actions as quickly as possible. Despite intense research on machine learning for the prediction of clinical outcomes, the acceptance of the integration of such complex models in clinical routine remains unclear. The aim of this study was to evaluate user acceptance of an already implemented machine learning-based application predicting the risk of delirium for in-patients. We applied a mixed methods design to collect opinions and concerns from health care professionals including physicians and nurses who regularly used the application. The evaluation was framed by the Technology Acceptance Model assessing perceived ease of use, perceived usefulness, actual system use and output quality of the application. Questionnaire results from 47 nurses and physicians as well as qualitative results of four expert group meetings rated the overall usefulness of the delirium prediction positively. For healthcare professionals, the visualization and presented information was understandable, the application was easy to use and the additional information for delirium management was appreciated. The application did not increase their workload, but the actual system use was still low during the pilot study. Our study provides insights into the user acceptance of a machine learning-based application supporting delirium management in hospitals. In order to improve quality and safety in healthcare, computerized decision support should predict actionable events and be highly accepted by users.

Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study.

OBJECTIVE: Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest-based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting. MATERIALS AND METHODS: Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting. RESULTS: During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r = 0.81) and nonblinded (r = 0.62) settings. A major advantage of our setting was the timely prediction without additional data entry. DISCUSSION: The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals. CONCLUSIONS: Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.

Multivariable Risk Prediction of Dysphagia in Hospitalized Patients Using Machine Learning.

BACKGROUND: Dysphagia is a dysfunction of the swallowing act and is highly prevalent in acute post-stroke patients and patients with chronic neurological diseases. Dysphagia is associated with several potentially life threatening complications. Thus, an early identification and treatment could reduce morbidity and mortality rates. OBJECTIVES: The aim of the study was to develop a multivariable model predicting the individual risk of dysphagia in hospitalized patients. METHODS: We trained different machine learning algorithms on the electronic health records of over 33,000 patients. RESULTS: The tree-based Random Forest Classifier and Adaboost Classifier algorithms achieved an area under the receiver operating characteristic curve of 0.94. CONCLUSION: The developed models outperformed previously published models predicting dysphagia. In future, an implementation in the clinical workflow is needed to determine the clinical benefit.

Development of a Machine Learning Model Predicting an ICU Admission for Patients with Elective Surgery and Its Prospective Validation in Clinical Practice.

Frequent utilization of the Intensive Care Unit (ICU) is associated with higher costs and decreased availability for patients who urgently need it. Common risk assessment tool, like the ASA score, lack objectivity and do account only for some influencing parameters. The aim of our study was (1) to develop a reliable machine learning model predicting ICU admission risk after elective surgery, and (2) to implement it in a clinical workflow. We used electronic medical records from more than 61,000 patients for modelling. A random forest model outperformed other methods with an area under the curve of 0.91 in the retrospective test set. In the prospective implementation, the model achieved a sensitivity of 73.3% and a specificity of 80.8%. Further research is essential to determine physicians' attitudes to machine learning models and assess the long term improvement of ICU management.

An Improvised Classification Model for Predicting Delirium.

With the vast increase of digital healthcare data, there is an opportunity to mine the data for understanding inherent health patterns. Although machine-learning techniques demonstrated their applications in healthcare to answer several questions, there is still room for improvement in every aspect. In this paper, we are demonstrating a method that improves the performance of a delirium prediction model using random forest in combination with logistic regression.

Information Adapted Machine Learning Models for Prediction in Clinical Workflow.

BACKGROUND: In a database of electronic health records, the amount of available information varies widely between patients. In a real-time prediction scenario, a machine learning model may receive limited information for some patients. OBJECTIVES: Our aim was to evaluate the influence of missing data on real-time prediction of delirium, and detect changes in prediction performance when training separate models for patients with missing data. METHODS: We compared a model trained specifically on data with missing values to the currently implemented model predicting delirium. Also, we simulated five test data sets with different amount of missing data and compared the prediction results to the prediction on complete data set when using the same model. RESULTS: For patients with missing laboratory and nursing assessment data, a model trained especially for this scenario performed significantly better than the implemented model. The combination of procedure data and demographic data achieved the closest results to a prediction with a complete data set. CONCLUSION: An ongoing evaluation of real-time prediction is indispensable. Additional models adapted to the information available might improve prediction performance.

Is Regular Re-Training of a Predictive Delirium Model Necessary After Deployment in Routine Care?

Adoption of electronic medical records in hospitals generates a large amount of data. Health care professionals can easily lose their sight on the important insights of the patients' clinical and medical history. Although machine learning algorithms have already proved their significance in healthcare research, remains a challenge translation and dissemination of fully automated prediction algorithms from research to decision support at the point of care. In this paper, we address the effect of changes in the characteristics of data over time on the performance of deployed models for the use case of predicting delirium in hospitalised patients. We have analysed the stability of models trained with subsets of data from one single year (2012, 2013...2016, respectively), and tested the models with data from 2017. Our results show that in the case of delirium prediction, the models were stable over time, indicating that re-training the models is not necessary e.g. once per year might be more than sufficient.

Risk assessment for preterm preeclampsia in first trimester: Comparison of three calculation algorithms.

OBJECTIVE: To better adjust the risk for preeclampsia, multifactorial models in first trimester of pregnancy have found the way in clinical practice. This study compares the available test algorithms. STUDY DESIGN: In a cross-sectional study between November 2013 and April 2016 we compared the tests results of three first trimester testing algorithms for preeclampsia in 413 women. Risk for preterm preeclampsia was calculated with three different algorithms: Preeclampsia Predictor™ Software by PerkinElmer (PERK), ViewPoint® Software by GE Healthcare (VP) and the online calculator of the Fetal Medicine Foundation (FMF).We analyzed the data descriptively and determined Cohen's Kappa to assess the agreement among the algorithms. RESULTS: VP classified 89(21.5%) women, PERK 43(10.4%) women and FMF 90 (21.8%) women as having high risk for preterm preeclampsia (<34 weeks of gestation for VP and PERK and <37 weeks of gestation for FMF). Agreement between tests ranged from moderate to substantial (PERK/VP: κ = 0.56, PERK/ FMF: κ = 0.50, and VP/ FMF: κ = 0.72). CONCLUSION: The three algorithms are similar but not equal. This may depend on chosen cut off, but also on test properties. This study cannot decide which algorithm is the best, but differences in results and cut offs should be taken into account.

On the Representation of Machine Learning Results for Delirium Prediction in a Hospital Information System in Routine Care.

Digitalisation of health care for the purpose of medical documentation lead to huge amounts of data, hence having an opportunity to derive knowledge and associations of different attributes recorded. Many health care events can be prevented when identified. Machine learning algorithms could identify such events but there is ambiguity in understanding the suggestions especially in clinical setup. In this paper we are presenting how we explain the decision based on random forest to health care professionals in the course of the project predicting delirium during hospitalisation on the day of admission.

Evaluating the Impact of Incorrect Diabetes Coding on the Performance of Multivariable Prediction Models.

The use of electronic health records for risk prediction models requires a sufficient quality of input data to ensure patient safety. The aim of our study was to evaluate the influence of incorrect administrative diabetes coding on the performance of a risk prediction model for delirium, as diabetes is known to be one of the most relevant variables for delirium prediction. We used four data sets varying in their correctness and completeness of diabetes coding as input for different machine learning algorithms. Although there was a higher prevalence of diabetes in delirium patients, the model performance parameters did not vary between the data sets. Hence, there was no significant impact of incorrect diabetes coding on the performance for our model predicting delirium.

Cleansing and Imputation of Body Mass Index Data and Its Impact on a Machine Learning Based Prediction Model.

BACKGROUND: A challenge of using electronic health records for secondary analyses is data quality. Body mass index (BMI) is an important predictor for various diseases but often not documented properly. OBJECTIVES: The aim of our study is to perform data cleansing on BMI values and to find the best method for an imputation of missing values in order to increase data quality. Further, we want to assess the effect of changes in data quality on the performance of a prediction model based on machine learning. METHODS: After data cleansing on BMI data, we compared machine learning methods and statistical methods in their accuracy of imputed values using the root mean square error. In a second step, we used three variations of BMI data as a training set for a model predicting the occurrence of delirium. RESULTS: Neural network and linear regression models performed best for imputation. There were no changes in model performance for different BMI input data. CONCLUSION: Although data quality issues may lead to biases, it does not always affect performance of secondary analyses.

Effect of Nursing Assessment on Predictive Delirium Models in Hospitalised Patients.

Delirium is an acute neuropsychiatric syndrome which is common in elderly patients during their hospitalisation and is associated with an increased mortality and morbidity. Since delirium is a) often underdiagnosed and b) preventable if early signs are detected,igh expectations are set in delirium risk assessment during hospital admission. In our latest studies, we showed that delirium prediction using machine learning algorithms is possible based on the patients' health history. The aim of this study is to compare the influence of nursing assessment data on prediction models with clinical and demographic data. We approached the problem by a) comparing the performance of predictive models including nursing data with models based on clinical and demographic data only and b) analysing the feature importance of all available features. From our results we concluded that nursing assessment data can improve the performance of delirium prediction models better than demographic, laboratory, diagnosis, procedures, and previous transfers' data alone.

Small airway function before and after cold dry air challenge in pediatric asthma patients during remission.

BACKGROUND: We wanted to compare cold dry air challenge (CACh) induced changes in spirometric parameters with changes in nitrogen multiple breath washout (N2 MBW) parameters in pediatric asthma patients during clinical remission over the past year (ie, with "inactive asthma"). As N2 MBW assesses ventilation heterogeneity we expected to gain detailed information about peripheral airways contribution. METHODS: In subjects with normal spirometry N2 MBW, spirometry and body plethysmography were performed at baseline, after CACh, and after salbutamol inhalation. An initial measurement of the fraction of exhaled nitric oxide (FeNO) was conducted. RESULTS: Forty-three (20 female) subjects, mean age 13.7 years (range 6.5-18.6) performed reproducible N2 MBW measurements. Ten were tested hyperresponsive (23.3%) and 33 normoresponsive (76.7%). Baseline spirometry and body plethysmography as well as FRC (N2 MBW) were similar in both groups. Scond (0.031 vs 0.022), Sacin (0.057 vs 0.067), and FeNO (92.0 vs 28.5 ppb) were not statistically different between hyperresponsive and nomoresponsive subjects at baseline. Subjects with airway hyperresponsiveness (AHR) showed significant increases in lung clearance index (LCI, P = 0.011) and Scond (P = 0.008) after CACh, and significant decreases after salbutamol (LCI: P = 0.005; Scond: P = 0.005). In contrast, normoresponsive subjects showed no relevant changes after CACh, and only a decrease of Scond after salbutamol (P = 0.007). There were significant correlations between the CACh induced changes in FEV1 and changes in LCI (r = -0.45, P = 0.003), Scond (r = -0.30, P = 0.047), and Sacin (r = -0.47, P = 0.008), respectively. CONCLUSION: Our study provides evidence of small airway involvement in children and adolescents with inactive asthma and airway hyperresponsiveness.

A retrospective study of isolated fractures of the alveolar process in the permanent dentition.

BACKGROUND/AIM: There is a lack of studies of fractures of the alveolar process (FAP). Only five were published in the last 50 years. The aim of this study was to analyze the risk of pulp necrosis and infection (PN), pulp canal obliteration (PCO), infection-related root resorption (IRR), ankylosis-related resorption (ARR), marginal bone loss (MBL), and tooth loss (TL) as well as to identify the possible risk factors for teeth involved in an isolated alveolar process fracture. In the second part, any late complications of the involved teeth were reported in patients who responded to a follow-up examination. MATERIAL AND METHOD: This study was a retrospective analysis of 126 patients with 329 traumatized permanent teeth treated in a regional dental trauma clinic. Follow-up examination was performed on 31 (24.6%) patients with 75 (22.8%) teeth. The risks of PN, PCO, RR, MBL, and TL were analyzed using the Kaplan-Meier method. Possible risk factors for PN (stage of root development, fracture position in relation to the root apex, concomitant injury, treatment delay, and antibiotics) were analyzed using univariate and multivariate Cox regression and generalized estimating equation. The level of significance was 5%. RESULTS: Pulp necrosis was observed in 43% of the teeth, and it was significantly associated with the presence of a concomitant injury and complete root formation. PCO was recorded in 2.8%, root resorption (RR, IRR, and ARR) in 4%, MBL in 8%, and TL in 0.6% of the teeth. Thirty-four percent of the teeth were assumed to have normal pulps, but they did not respond to pulp sensibility testing. At the follow-up examination, PN was found in 49%, PCO in 28%, RR (IRR and ARR) in 4%, MBL in 17%, and TL in 5%. Estimated risk after a 5-years follow up was as follows: PN: 48.2% (95% confidence interval (CI): 42.0-54.5), IRR: 7.2 (95% CI: 3.5-10.9), ARR: 33.0% (95% CI: 22.4-43.6), BL: 16.7% (95% CI: 9.6-23.8), TL: 4.0% (95% CI: 0.0-8.5). The following factors significantly increased the risk of PN: mature root development (hazard ratio [HR]: 7.50 [95% CI: 1.84-30.64], P=.005) and concomitant injury (HR: 2.68 [95% CI: 1.76-4.09], P<.001). In a logistic regression model, teeth with mature roots had a threefold risk of becoming non-responsive to pulp testing. CONCLUSION: Teeth involved in an isolated alveolar process fracture and managed with a conservative treatment approach appear to have a good prognosis. The most common complication was PN which did not negatively affect the survival of the teeth after root canal treatment.